An FPGA-based Prototyping Platform for Research in High-Speed Interprocessor Communication

Parallel and multinode computing systems are becoming widespread and grow in sophistication. Besides simulation, rapid prototyping becomes important in designing and evaluating their architecture. We present an FPGA-based system that we developed and use for prototyping and measuring high speed processor-network interfaces and interconnects; it is an experimental tool for research projects in architecture. We configure FPGA boards as network interfaces (NI) and as switches. NI’s plug into the PCI-X bus of commercial PC’s, and use 4 links of 2.5 Gb/s/link as network connections; we can bundle these links together, at the byte or packet level, offering 10 Gb/s of network throughput. NI’s implement DMA on the PCI-X side, and remote DMA and remote notification (interrupt or flag-setting) on the network side. We configured the switch boards as buffered crossbars operating directly on variable-size packets and featuring credit-based flow control for lossless communication. Multiple, parallel switches can serve the NI links using multipath routing; NI’s resequence the out-of-order packet arrivals. All boards provide extensive support for monitoring, debugging, and measurement. Colleagues adapted the Linux OS for this platform, and used it for remote disk I/O experiments [1]. We report here on the platform architecture, its design cost and complexity, latency and throughput parameters, and buffered crossbar performance. We now work on remote queues and synchronization mechanisms.